Effect of a doctor working during the festive period on population health:natural experiment using 60 years of Doctor Who episodes (the TARDIS study)

OBJECTIVE: To examine the effect of a (fictional) doctor working during the festive period on population health.DESIGN: Natural experiment.SETTING: England, Wales, and the UK.MAIN OUTCOME MEASURES: Age standardised annual mortality rates in England, Wales, and the UK from 1963, when the BBC first broadcast Doctor Who, a fictional programme with a character called the Doctor who fights villains and intervenes to save others while travelling through space and time. Mortality rates were modelled in a time series analysis accounting for non-linear trends over time, and associations were estimated in relation to a new Doctor Who episode broadcast during the previous festive period, 24 December to 1 January. An interrupted time series analysis modelled the shift in mortality rates from 2005, when festive episodes of Doctor Who could be classed as a yearly Christmas intervention.RESULTS: 31 festive periods from 1963 have featured a new Doctor Who episode, including 14 broadcast on Christmas Day. In time series analyses, an association was found between broadcasts during the festive period and subsequent lower annual mortality rates. In particular, episodes shown on Christmas Day were associated with 0.60 fewer deaths per 1000 person years (95% confidence interval 0.21 to 0.99; P=0.003) in England and Wales and 0.40 fewer deaths per 1000 person years (0.08 to 0.73; P=0.02) in the UK. The interrupted time series analysis showed a strong shift (reduction) in mortality rates from 2005 onwards in association with the Doctor Who Christmas intervention, with a mean 0.73 fewer deaths per 1000 person years (0.21 to 1.26; P=0.01) in England and Wales and a mean 0.62 fewer deaths per 1000 person years (0.16 to 1.09; P=0.01) in the UK.CONCLUSIONS: A new Doctor Who episode shown every festive period, especially on Christmas Day, was associated with reduced mortality rates in England, Wales, and the UK, suggesting that a doctor working over the festive period could lower mortality rates. This finding reinforces why healthcare provision should not be taken for granted and may prompt the BBC and Disney+ to televise new episodes of Doctor Who every festive period, ideally on Christmas Day.</p

Stability of clinical prediction models developed using statistical or machine learning methods

Clinical prediction models estimate an individual's risk of a particular health outcome, conditional on their values of multiple predictors. A developed model is a consequence of the development dataset and the chosen model building strategy, including the sample size, number of predictors and analysis method (e.g., regression or machine learning). Here, we raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Then, through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers should always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model building steps (those used in the development of the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and then deriving (i) a prediction instability plot of bootstrap model predictions (y-axis) versus original model predictions (x-axis), (ii) a calibration instability plot showing calibration curves for the bootstrap models in the original sample; and (iii) the instability index, which is the mean absolute difference between individuals' original and bootstrap model predictions. A case study is used to illustrate how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), whilst also informing a model's critical appraisal (risk of bias rating), fairness assessment and further validation requirements.Comment: 30 pages, 7 Figure

Developing clinical prediction models when adhering to minimum sample size recommendations: The importance of quantifying bootstrap variability in tuning parameters and predictive performance

Recent minimum sample size formula (Riley et al.) for developing clinical prediction models help ensure that development datasets are of sufficient size to minimise overfitting. While these criteria are known to avoid excessive overfitting on average, the extent of variability in overfitting at recommended sample sizes is unknown. We investigated this through a simulation study and empirical example to develop logistic regression clinical prediction models using unpenalised maximum likelihood estimation, and various post-estimation shrinkage or penalisation methods. While the mean calibration slope was close to the ideal value of one for all methods, penalisation further reduced the level of overfitting, on average, compared to unpenalised methods. This came at the cost of higher variability in predictive performance for penalisation methods in external data. We recommend that penalisation methods are used in data that meet, or surpass, minimum sample size requirements to further mitigate overfitting, and that the variability in predictive performance and any tuning parameters should always be examined as part of the model development process, since this provides additional information over average (optimism-adjusted) performance alone. Lower variability would give reassurance that the developed clinical prediction model will perform well in new individuals from the same population as was used for model development

Performance of methods for meta-analysis of diagnostic test accuracy with few studies or sparse data

Hierarchical models such as the bivariate and hierarchical summary receiver operating characteristic (HSROC) models are recommended for meta-analysis of test accuracy studies. These models are challenging to fit when there are few studies and/or sparse data (for example zero cells in contingency tables due to studies reporting 100% sensitivity or specificity); the models may not converge, or give unreliable parameter estimates. Using simulation, we investigated the performance of seven hierarchical models incorporating increasing simplifications in scenarios designed to replicate realistic situations for meta-analysis of test accuracy studies. Performance of the models was assessed in terms of estimability (percentage of meta-analyses that successfully converged and percentage where the between study correlation was estimable), bias, mean square error and coverage of the 95% confidence intervals. Our results indicate that simpler hierarchical models are valid in situations with few studies or sparse data. For synthesis of sensitivity and specificity, univariate random effects logistic regression models are appropriate when a bivariate model cannot be fitted. Alternatively, an HSROC model that assumes a symmetric SROC curve (by excluding the shape parameter) can be used if the HSROC model is the chosen meta-analytic approach. In the absence of heterogeneity, fixed effect equivalent of the models can be applied

Deriving percentage study weights in multi-parameter meta-analysis models: with application to meta-regression, network meta-analysis and one-stage individual participant data models.

Many meta-analysis models contain multiple parameters, for example due to multiple outcomes, multiple treatments or multiple regression coefficients. In particular, meta-regression models may contain multiple study-level covariates, and one-stage individual participant data meta-analysis models may contain multiple patient-level covariates and interactions. Here, we propose how to derive percentage study weights for such situations, in order to reveal the (otherwise hidden) contribution of each study toward the parameter estimates of interest. We assume that studies are independent, and utilise a decomposition of Fisher's information matrix to decompose the total variance matrix of parameter estimates into study-specific contributions, from which percentage weights are derived. This approach generalises how percentage weights are calculated in a traditional, single parameter meta-analysis model. Application is made to one- and two-stage individual participant data meta-analyses, meta-regression and network (multivariate) meta-analysis of multiple treatments. These reveal percentage study weights toward clinically important estimates, such as summary treatment effects and treatment-covariate interactions, and are especially useful when some studies are potential outliers or at high risk of bias. We also derive percentage study weights toward methodologically interesting measures, such as the magnitude of ecological bias (difference between within-study and across-study associations) and the amount of inconsistency (difference between direct and indirect evidence in a network meta-analysis)

Two-stage or not two-stage? That is the question for IPD meta-analysis projects

Individual participant data meta-analysis (IPDMA) projects obtain, check, harmonise and synthesise raw data from multiple studies. When undertaking the meta-analysis, researchers must decide between a two-stage or a one-stage approach. In a two-stage approach, the IPD are first analysed separately within each study to obtain aggregate data (e.g., treatment effect estimates and standard errors); then, in the second stage, these aggregate data are combined in a standard meta-analysis model (e.g., common-effect or random-effects). In a one-stage approach, the IPD from all studies are analysed in a single step using an appropriate model that accounts for clustering of participants within studies and, potentially, between-study heterogeneity (e.g., a general or generalised linear mixed model). The best approach to take is debated in the literature, and so here we provide clearer guidance for a broad audience. Both approaches are important tools for IPDMA researchers and neither are a panacea. If most studies in the IPDMA are small (few participants or events), a one-stage approach is recommended due to using a more exact likelihood. However, in other situations, researchers can choose either approach, carefully following best practice. Some previous claims recommending to always use a one-stage approach are misleading, and the two-stage approach will often suffice for most researchers. When differences do arise between the two approaches, often it is caused by researchers using different modelling assumptions or estimation methods, rather than using one or two stages per se

Determinants of Childhood Adiposity: Evidence from the Australian LOOK Study

BACKGROUND To contribute to the current debate as to the relative influences of dietary intake and physical activity on the development of adiposity in community-based children. METHODS Participants were 734 boys and girls measured at age 8, 10 and 12 years for percent body fat (dual emission x-ray absorptiometry), physical activity (pedometers, accelerometers); and dietary intake (1 and 2-day records), with assessments of pubertal development and socioeconomic status. RESULTS Cross-sectional relationships revealed that boys and girls with higher percent body fat were less physically active, both in terms of steps per day and moderate and vigorous physical activity (both sexes p<0.001 for both measures). However, fatter children did not consume more energy, fat, carbohydrate or sugar; boys with higher percent body fat actually consumed less carbohydrate (p = 0.01) and energy (p = 0.05). Longitudinal analysis (combined data from both sexes) was weaker, but supported the cross-sectional findings, showing that children who reduced their PA over the four years increased their percent body fat (p = 0.04). Relationships in the 8 year-olds and also in the leanest quartile of all children, where adiposity-related underreporting was unlikely, were consistent with those of the whole group, indicating that underreporting did not influence our findings. CONCLUSIONS These data provide support for the premise that physical activity is the main source of variation in the percent body fat of healthy community-based Australian children. General community strategies involving dietary intake and physical activity to combat childhood obesity may benefit by making physical activity the foremost focus of attention.The financial support provided by the Commonwealth Education Trust (London, UK) was vital to the completion of this work, and the authors thank the Board of Trustees for supporting them over several years. The authors also thank members of The Canberra Hospital Salaried Staff Specialists Private Practice Fund for their financial contribution to the study. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript